fort monroe

TRADOC Quality Assurance

Program

External Evaluation

by Rosanne May

TRADOC Quality Assurance Office

Victory Through Excellent

External Evaluation Process

Sourc e s

• CALL

• CTC Rotation Feedback

• Lessons Learned Collectors

• Surveys

• TD/CD/DD Developers Feedback

• Conferences/seminars

• Field Visits

• Unit Feedback

• Accident Reports

• Unsolicited feedback

• Internal evaluation results

Survey Process

Survey Operational Concept

• Survey all courses over three-year cycle

• Use team concept (QA/TD/T collaborate)

• Consider characteristics of survey population

- Number of graduates

- Where assigned

- How to reach supervisors

Identify survey marketing requirements

Survey Planning

• Develop MEP (Master Evaluation Plan)

- Schedule surveys over a three-year cycle or when course is revised
(> 30 %)

- Coordinate proposed schedule within the school

- Obtain approval of the coordinated schedule

- Include the schedule in Master Evaluation Plan (MEP)

- Obtain commandant approval of the MEP

- Include the MEP data as part of the Proponent Training Development
Plan

• Submit approved MEP to TRADOC QAO by 1 September

• Plan survey for a specific course. Determine —

- who gets survey results

- what is target population size

- what level of reliability is needed

- What size sample is needed for desired reliability

- what additional questions to add to survey

- how long to field survey

• Develop External Evaluation Project Management Plan

Developing and Collecting Data

Develop survey

- Obtain list of AKO email addresses for course graduates

- Coordinate specific requirements for survey questions with A

- Obtain Excel files (easiest to input) for approved -
o Course subject areas/subject area descriptions
o List of task titles (grouped by subject area)
o Additional questions

- Assemble the survey using the AUTOGEN program

- Generate survey in all the formats selected (e.g., email, web)

- Have SME review and revise survey accordingly

Field the survey.

- Post survey on the ARI server

- Send emails requesting responders to take survey (via ARI/proponent school
server)

- Set up computers station at in-processing for LD courses

Verify survey is available

Track survey response rates

Send a periodic reminder email to survey sample

•Determine if sample size is large enough to provide reliable
data

•Compile survey results

- Select and run required preset AUTOGEN reports

- Design and run customized reports.

- Determine if needed reports exceed the capabilities of AUTOGEN

- Extract data from AUTOGEN database to perform additional
analysis

•Analyze data with SME and course manager input

- Identify deficiencies found

- Determine causes and develop solutions

- Identify positive indicators, supporting factors, and responsible
organizations

•Compare results to of this survey with other sources to
identify

trends

Report F i nd i ng and —

R e port

- Identify trends, positive indicators, supporting factors, ar f Report

, i findings}

responsible \ recommtn

organizations

- Report course deficiencies and recommend specific solution
include

responsible organization

- Identify trends that extend across courses, to include positive
indicators

that should be promulgated as well as deficiencies and
recommended
solutions

Collaborate on a solution set with DOTD and other
stakeholders

Brief the commander/commandant on agreed upon
solution set

Follow Up

Identify —

• Which approved solutions have been implemente

• Which solutions have not been implemented V

• What is the timeline for implementing remaining solutions

• If there are any changes in who is responsible for implementing
solutions

• What issues are inhibiting implementation of approved solutions

What actions QAO/QAE can take to facilitate implementation

Summary